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ABSTRACT 

Cognitive  Workload  in  Fault  Diagnosis 


Prior  research  has  produced  a  capability  to  project  with  reasonable  accuracy  the  manual 
activities  required  to  accomplish  diagnosis  and  repair  of  particular  faults  within  a  specified  system. 
When  applied  to  a  substantial  sample  of  faults,  the  technique,  termed  PROFILE,  provides  an 
assessment  of  the  manual  workload  expected  to  maintain  the  system. 

To  obtain  usable  projections  of  total  repair  times,  however,  requires  the  estimation  of  the 
cognitive  portion  of  fault  diagnosis  time  as  well.  Unfortunately,  the  cognitive  processes  involved 
in  fault  diagnosis  are  complex,  variable  from  one  individual  and  circumstance  to  another,  and 
almost  entirely  unsupported  with  research  data  quantifying  their  execution  times.  It  becomes 
necessary,  therefore,  to  explore  the  variations  in  cognitive  times  in  relation  to  those  characteristics 
of  the  diagnostic  situation  which  can  be  quantified.  Three  such  characteristics  are  analyzed  in 
detail:  I)  the  number  of  tests  required  to  isolate  a  fault,  2)  the  total  manual  time  expended  in  fault 
diagnosis,  and  3)  the  times  of  the  individual  tests  in  a  diagnostic  sequence. 

Previous  studies  involving  87  technical  participants,  and  three  different  systems  under  repair, 
obtained  detailed  data  on  638  fault  diagnosis  episodes.  Statistical  analyses  of  these  performance 
data  reveal  that  the  average  cognitive  time  expended  per  fault  was  related  to  a  significant  degree  to 
the  three  diagnostic  characteristics.  The  most  reliable  predictor  of  actual  cognitive  time  for  a 
particular  fault  was  obtained  by  computing  an  estimate  cognitive  time  prior  to  each  test  based  upon 
the  manual  time  required  to  perform  that  test  The  function  relating  cognitive  time  to  the  manual 
time  of  the  following  test  action  is  a  non-linear  function  derived  from  the  experimental  data  It 
indicates  that  cognitive  time  increases  when  the  manual  time  of  the  following  test  increases,  but 
quickly  approaches  an  upper  limit. 

Since  the  PROFILE  troubleshooting  model  produces  good  projections  of  the  particular  tests 
performed  to  accomplish  diagnosis  and  repair,  and  their  time  durations,  it  can  now  dso  compute 
the  estimated  cognitive  time  preceding  each  test 

The  sum  of  the  projected  cognitive  and  manual  times  correspond  well  with  the 
experimentally  observ^  total  diagnosis  and  repair  times. 
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SECTION  I.  INTRODUCTION 


Research  conducted  over  the  past  four  years  has  led  to  techniques  for  projecting  the  manual 
workload  involved  in  performing  corrective  maintenance  upon  specific  systems.  The  ultimate 
objective  is  to  develop  a  process  for  projecting  the  distribution  of  total  maintenance  times  (manual 
and  cognitive  components)  for  a  system  based  upon  its  design  specifications.  Such  a  resource 
could  support  the  design  team  in  evaluating  alternative  design  options,  and  it  could  be  useful  in 
procurement  and  logistic  activities  as  well. 

The  approach  is  'performance-based',  i.e.,  the  computed  time  to  diagnose  and  correct  a 
particular  fault  is  synthesized  from  a  detailed  projection  of  the  maintenance  tasks  required  to 
accomplish  isolation  and  repair  of  the  fault.  Doing  this  over  a  sample  of  faults  in  a  system 
produces  a  distribution  of  maintenance  times  which  characterize  the  maintainability  of  the  design. 

Earlier  reports  describe  the  development  of  the  computer  model  of  an  expert  troubleshooter 
(Towne,  Johnson,  and  Corwin,  1982,  1983;  Towne  and  Johnson,  1984).  The  model,  called 
PROFILE,  generates  a  sequence  of  manual  actions  to  restore  a  system  containing  a  specified 
failure.  The  manual  operations  which  might  be  generated  for  a  particular  fault  include  testing, 
adjusting,  disassembling,  replacing,  and  reassembling.  Each  of  the  manual  actions  can  be  assigned 
a  representative  performance  time,  using  traditional  pre-determined  time  techniques  (Karger  & 

Bay  ha,  1966).  The  total  manual  times  projected  by  PROFILE  compare  well  with  observed  manual 
times. 


Three  important  results  of  this  performance-based  approach  are  1)  it  is  sensitive  to  the  ways 
in  which  the  internal  organization  of  the  system  affects  the  fault  isolation  process,  2)  it  is  sensitive 
to  the  ways  in  which  the  physical  structure  of  the  system  affects  the  necessary  disassembly,  testing, 
and  repair  tasks,  and  3)  it  provides  the  ability  to  identify  the  source  of  maintainability  problems, 
and  to  quantify  their  effects,  all  within  the  design  phase  of  system  development 

Cognitive  Aspects  of  Fault  Diagnosis 

The  present  goal  is  to  generate  representative  projections  of  the  cognitive  workload  in  fault 
diagnosis,  measured  in  time  units.  The  power  of  the  performance-based  approach  is  essential  to 
this  endeavor,  for  it  provides  a  reasonably  accurate  account  of  the  diagnostic  process  associated 
with  a  particular  failure. 

Fault  diagnosis  is  an  activity  which  can  present  a  fascinating  and  somewhat  unique  mental 
challenge.  In  the  domain  of  everyday  experience,  troubleshooting  seems  to  lie  somewhere  between 

1 


solving  a  crime  and  disarming  a  bomb.  As  the  diagnostician  initiates  an  investigation,  he/she  is 
presented  a  report  that  some  undesirable  events  have  occurred,  resulting  in  an  improperly  operating 
physical  system.  The  report  may  be  fully  reliable,  partially  incorrect,  or  completely  erroneous. 
Further  evidence  may  be  gathered,  but  often  the  fault  cannot  be  directly  observed.  The 
diagnostician  usually  has  the  option  of  proceeding  to  gather  further  evidence  or  of  devoting  time  to 
further  consideration  of  the  available  information. 

To  further  complicate  the  situation,  the  actions  of  the  diagnostician  may  increase  the  level  of 
damage,  they  may  present  a  danger  to  the  investigator  or  to  the  system  under  repair,  and  sometimes 
they  remedy  the  fault  in  ways  that  forever  disguise  its  true  nature.  In  the  worst  cases,  there  are 
multiple  faults  or  faults  which  vary  over  time  in  ways  which  are  exceedingly  difficult  to  anticipate 
and  fully  understand. 

The  full  inventory  of  cognitive  functions  involved  in  fault  diagnosis  entails  an  extremely 
wide  range  of  mental  activities.  The  diagnostician  may  have  to  recall  recent  events  (symptoms 
obtained  and  test  configurations  employed)  as  well  as  events  from  the  distant  past  (prior  faults 
experienced,  and  their  symptoms).  The  troubleshooter  may  have  to  recall  or  generate  theories 
concerning  how  the  particular  system  operates  in  various  conditions.  This  may  require  reasoning 
about  the  possible  cause  of  disperse  symptom  information,  and  about  the  specific  tests  which 
would  discriminate  those  possibilities.  Finally,  the  troubleshooter  must  accomplish  some  type  of 
self-direction,  ranging  from  conscious  and  explicit  consideration  of  the  available  resources  and 
constraints,  to  apparently  ad-hoc  decisions  which  the  individual  might  have  difficulty  explaining. 

While  there  have  been  considerable  investigations  into  the  nature  of  problem  solving  and 
planning  processes,  the  focus  has  been  primarily  upon  the  types  of  functions  and  activities 
performed,  and  the  characteristics  of  that  mental  performance,  rather  than  the  time  required.  Hamm 
(1985)  has  studied  the  ways  in  which  experts  shift  their  cognitive  effort  between  'intuitive  and 
analytical  cognition';  Rouse  and  Morris  (1985)  have  considered  the  relationships  between  the 
domain  and  possible  mental  models  held  by  human  problem  solvers;  and  Kahneman  and  Tversky 
(1979)  have  looked  deeply  into  the  ways  human  beings'  decisions  are  influenced  by  risk.  Perhaps 
the  most  pertinent  previous  r;scarch  has  been  done  by  Hayes-Roth  (1980)  in  studying  the  ways  in 
which  people  plan  complex  activities,  and  the  types  of  limitations  encountered  in  estimating  the  time 
required  to  accomplish  them. 

Considerable  work  has  been  done  measuring  the  time  requirements  to  perform  basic 
computation  and  perception  functions.  One  comprehensive  body  of  data  developed  by  Quick, 
(Quick,  Duncan,  &  Malcolm ,  1962)  provides  detailed  time  data  for  such  functions  as  visual 
inspection,  reacting,  computing,  and  reading. 


Efforts  to  quantify  cognitive  activity  in  fault  diagnosis  face  two  imposing  requirements:  first, 
to  make  reasonably  accurate  projections  of  the  cognitive  processes  involved  in  the  diagnosis  of  a 
particular  fault,  and  second,  to  quantify  the  time  workload  of  those  processes.  These  requirements 
are  well  beyond  the  current  state  of  the  art  in  cognitive  science.  Yet,  there  remains  a  clear  and 
present  need  to  arrive  at  assessments  of  cognitive  time  workload,  to  support  the  projection  of  total 
corrective  maintenance  times. 

Approach 

The  approach  taken  in  this  research  was  to  search  for  reliable  relationships  between  cognitive 
times  occurring  within  experimentally  observed  corrective  maintenance  operations  and  the 
knowable  characteristics  of  the  associated  task  sequences.  Projected  cognitive  times  are  then 
computed  by  applying  these  same  relationships  to  PROFTLE-generated  troubleshooting  sequences. 

Four  sections  follow.  Section  II  evaluates  of  the  relationships  between  experimentally 
observed  cognitive  times  and  the  nature  of  the  diagnostic  functions  which  were  performed.  The 
intent  of  this  section  is  to  present  the  findings,  both  positive  and  negative,  which  heavily  influenced 
the  search  for  a  generalized  approach  to  projecting  cognitive  time  in  troubleshooting. 

Section  in  explores  the  relationships  between  the  actual  cognitive  time  expenditures  and 
characteristics  of  the  PROFILE  projections  of  those  diagnostic  activities.  The  variables  analyzed 
here  arc  1 )  number  of  PROFILE  tests  to  isolate  a  fault,  2)  total  PROFILE  manual  time  to  isolate  a 
fault,  and  3)  the  times  of  the  individual  tests  projected  by  PROFILE. 

Section  IV  explores  cognitive  complexity  in  fault  diagnosis,  while  Section  V  summarizes  the 
findings  of  this  research  and  discusses  implications  for  future  research. 


SECTION  II.  COGNITIVE  TIME  WORKLOAD  IN  FAULT  DIAGNOSIS 


While  we  may  not  have  a  clear  idea  what  decisions  are  made  in  an  environment  as  complex 
as  fault  diagnosis,  it  may  be  possible  to  quantify  the  significant  factors  which  affect  decision  lime 
by  analyzing  data  from  actual  diagnostic  tasks  in  terms  of  the  characteristics  of  those  tasks. 

The  cognitive  time  data  used  for  this  work  were  collected  in  studies  conducted  over  the  last 
four  years.  The  primary  purpose  of  these  earlier  studies  was  to  obtain  samples  of  representative 
diagnostic  performance  for  use  in  deriving  theories  of  diagnostic  performance  and  in  testing  the 
PROFILE  model  as  it  evolved.  Three  basic  studies  were  conducted,  involving  a  total  of  87 
technicians,  three  different  types  of  failed  system  ,  and  30  different  faults,  as  shown  below  . 


Study  Technician  No.  Diagnostic 

No.  Participants  Faults  Episodes  Failed  System 


1 

48 

8 

384 

video  simulation  of  microcomputer  system 

2 

29 

6 

174 

graphic  simulation  of  network  diagram 

3 

5 

8 

40 

infrared  (IR)  transmitler/receiver,  design  A 

1 

S 

4Q 

infrared  (IR)  transmitter/receiver,  design  B 

Total 

87 

30 

638 

Table  1.  The  Studies  of  Diagnosis  Performance 

In  these  studies,  each  technician  diagnosed  and  'repaired'  each  of  the  faults.  In  study  one  the 
performance  of  the  tests,  adjustments,  and  replacements  selected  by  the  technicians  were  displayed 
on  a  color  video  screen,  in  real  time;  in  study  two  the  test  results  were  displayed  on  a  graphics 
computer  screen.  In  study  three  the  participants  performed  all  the  troubleshooting  work  on  real 
equipment,  except  for  the  replacements,  which  were  done  by  an  experimenter  at  the  request  of  the 
technician.  The  order  of  presentation  of  faults  to  individual  technicians  was  fully  counterbalanced. 
Full  details  of  the  three  systems  and  the  study  conditions  are  given  in  Towne  and  Johnson,  1984. 

In  all,  detailed  data  for  638  fault  isolation  and  repair  episodes  were  obtained,  each  consisting 
of  the  following  information: 

a.  the  sequence  of  tests,  adjustments,  and  replacements  performed 

b.  the  manual  time  expended  performing  each  test 

c.  the  time  period  between  each  manual  action,  termed  inter-step  cognitive  time. 


The  total  cognitive  time  for  a  problem,  bv  an  individual  technician,  is  the  sum  of  the 
inter-step  cognitive  times  for  that  individual's  diagnosis  procedure.  The  average  cognitive  time  for 
a  particular  problem  is  the  simple  mean  of  the  cognitive  times  for  the  individual  participants  on  that 
problem. 

According  to  the  definition  given  above,  cognitive  time  is  that  time  during  which  no  manual 
activity  is  being  performed.  It  is  recognized  that  cognitive  activities  also  can  occur  during  the 
performance  of  manual  actions.  These  could  range  from  monitoring  the  performance  of  the  manual 
actions  in  progress  to  platming  future  testing,  or  reasoning  about  results  already  obtained.  For  the 
purpose  of  projecting  total  maintenance  time,  however,  it  is  most  convenient  o  classify  work  as 
'manual'  when  observable  work  is  being  performed,  and  'cognitive'  when  no  manual  actions  are  in 
progress.  The  PROFILE  model  and  its  associated  manual  time  data  address  the  time  expenditures 
during  the  p)ortions  defined  as  manual,  even  if  some  cognitive  activities  occur  during  these  periods. 

System-level  Results 

Appendix  A  presents  the  experimentally  observed  time  data  for  the  three  systems  studied, 
accumulated  by  technician;  appendix  B  presents  the  results  accumulated  by  problem.  The  two 
questions  of  concern  here  are  whether  there  are  well-ordered  relationship>s  among  cognitive 
variables  within  a  particular  system,  and  whether  there  are  relationships  that  seem  to  hold  across 
systems. 

Table  2  lists  correlations  for  three  pairs  of  variables,  for  each  of  the  three  systems  studied. 
The  data  analyzed  here  are  the  individual  problem  solutions,  thus  for  example  the  sample  size  for 
system  one  is  384  (forty-eight  technicians,  eight  problems  per  technician). 


SYSTEM 


1 

2 

3 

Variable  Pair 

n=384 

n=174 

9 

11 

00 

o 

cognitive  time  to  solution 

versus 

number  of  steps  to  solution 

0.67 

0.67 

0.75 

cognitive  time  to  solution 

versus 

manual  time  to  solution 

0.61 

0.64 

cognitive  time  to  solution 

versus 

cognitive  time  jpcr  step 

0.80 

0.69 

0.65 

Table  2.  Correlations  at  the  Systems  Level 


The  correlations  given  in  table  2  are  all  significant.  Since  the  PROFILE  model  can  be  used 
to  estimate  the  number  of  steps  to  solution  and  the  manual  time  to  solution,  these  initially  appear  to 
be  promising  indicators  of  cognitive  time  workload.  The  major  concern  then  becomes  whether  the 
quantitative  relationships  are  consistent  across  system  designs.  Section  III  presents  the  results  of 
predicting  cognitive  time  in  this  manner. 

The  significant  correlations  within  systems  are  both  interesting  and  intuitively  reasonable. 
Diagnostic  sequences  requiring  more  testing  steps  or  higher  cognitive  time  per  step  would  naturally 
tend  to  be  higher  in  total  cognitive  time.  Furthermore,  solutions  which  require  higher  manual 
involvement,  either  as  a  result  of  increased  numbers  of  tests  or  the  necessity  to  perform  more 
lengthy  operations,  would  seem  to  also  involve  increased  cognitive  activity. 

The  low  correlation  between  cognitive  time  per  step  and  number  of  steps  is  more  difficult  to 
understand.  We  might  imagine  that  when  a  diagnostic  routine  requires  more  steps  to  solution  the 
cognitive  time  per  step  would  also  be  higher,  in  general,  as  the  problem  is  more  difficult 
Alternatively,  we  might  expect  a  subject-effect,  either  that  technicians  requiring  more  tests  to  find 
faults  also  are  slower  in  conducting  the  cognitive  actions,  or  that  those  individuals  who  devote  less 
time  to  non-manual  processes  suffer  by  having  to  perform  more  tests. 

Detailed  examination  of  individual  solution  sequences  reveals  the  possibility  of  several 
types  of  effects.  In  some  cases  technicians  appear  to  be  'fishing',  by  performing  many  tests,  but 
devoting  little  time  to  considering  the  results  or  to  selecting  the  next  test  Perhaps  these  individuals 
are  searching  for  some  particular  indication,  and  are  ignoring  the  more  subtle  information  content  of 
the  tests.  In  these  cases  a  high  number  of  tests  are  observed  with  a  low  cognitive  workload  per 
test.  There  are  other  cases  when  the  cognitive  times  per  test  are  unusually  high  when  the  number  of 
tests  to  isolation  is  high.  These  may  be  cases  where  the  symptom  information  is  difficult  to 
interpret,  where  the  selection  of  a  useful  test  is  difficult,  or  where  significant  strategic  alternatives 
are  available  to  be  considered.  They  may  also  be  cases  in  which  the  individual's  cognitive  style  is 
different  from  those  who  do  more  testing  and  less  evaluation  and  planning. 

Compounded  upon  these  partially  offsetting  effects  are  possible  problem  effects.  Some 
faults  may  lead  troubleshooters  into  extremely  complex  areas  of  the  system,  thereby  necessitating  a 
high  cognitive  involvement  per  test,  while  others  may  be  resolved  via  a  sequence  of  easily 
performed  and  interpreted  tests.  Qearly  the  detailed  performance  data  cannot  be  easily  interpreted 
unless  subject  effects  and  problem  effects  are  considered  individually.  The  next  section  looks 
deeper  into  the  variations  in  cognitive  time  attributable  to  individual  technicians,  while  the 
subsequent  section  investigates  problem  effects. 
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Individual  Differences  in  Cognitive  Time  Expenditures 


Do  better  diagnosticians  dwell  longer  on  the  meaning  of  tests  than  less  effective  technicians, 
or  do  they  spend  less  time,  possibly  as  a  result  of  better  understanding  or  innate  reasoning  power? 
Does  an  increased  investment  in  reasoning  time  pay  off  in  reduced  manual  time  and  reduced  total 
fault  isolation  time?  To  explore  these  questions  the  individual  technicians  are  ranked  according  to 
the  amount  of  cognitive  time  each  one  expended  solving  the  problems,  and  grouped  into  three 
categories  of  near-equal  size. 


Figure  1  presents  the  manual  and  cognitive  times  expended  by  each  group,  on  the  average, 
for  the  IR  system  (system  3).  While  the  cognidve  time  per  problem  varied  from  268  seconds  to 
503  seconds,  from  the  low  cognitive  group  to  the  high  cognitive  group,  the  manual  time  expended 
per  problem  was  virtually  identical  across  the  three  groups.  This  independence  between  manual 
time  and  cognitive  time,  at  the  individual  technician  level,  was  equally  true  for  the  other  two 
systems  studied.  An  analysis  of  variance  confirms  the  lack  of  a  significant  effect  of  subject-group 
upon  manual  time  (F=0.0529,  p=0.9485) 
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cognition  cognition  cognition 


Figure  1.  Cognitive  and  Manual  Times,  by  Cognitive  Group  -  IR  System 
(Technicians  Grouped  by  Cognitive  Time  per  Problem) 


Variations  in  number  of  tests  and  cognitive  time  per  test  may  be  examined  using  these  same 
groups.  Figure  2  presents  the  results  for  the  IR  System,  and  indicates  that  1)  the  number  of  tests 
performed  did  not  differ  significantly  by  cognitive  group,  and  2)  the  cognitive  tine  per  test 
increased  significandy  by  cognitive  group.  This  indicates  that  the  reason  for  the  increased  cognitive 
time  per  problem,  for  the  higher-cognition  group,  was  higher  cognitive  time  spent  per  test,  rather 
than  increased  numbers  of  tests. 


Appendix  C  provides  a  summary  of  these  data,  for  the  three  cognitive  groups  in  each  study. 
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Figure!  Number  of  Tests  and  Cognitive  Time  per  Test  -IR  System 
(Technicians  Grouped  by  Cognitive  Time  Per  Problem) 

When  individuals  are  grouped  by  their  total  isolation  and  repair  time,  including  both 
cognitive  and  manual  effort,  the  better  maintainers  are  found  to  expend  less  time  in  both  manual  and 
cognitive  activity  in  all  three  systems  studied,  with  the  one  exception  that  the  middle-total  group  in 
the  IR  System  study  performed  slightly  less  manual  work  than  the  lower-total  group.  Figure  3 
presents  the  results  for  the  IR  system. 
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Figure  3.  Cognitive  and  Manual  Times,  by  Cognitive  Group  -  IR  System 
(Technicians  Grouped  by  Total  Time  per  Problem) 

The  reduced  manual  time  expended  by  the  better  troubleshooters  appears  to  be  a  result  of 
better  test  selection,  rather  than  a  faster  workpace,  for  the  pace  of  performing  manual  tasks  was 
controlled  in  two  of  the  three  studies  (in  studies  one  and  two  all  manual  operations  selected  by  the 
subjects  were  shown  being  perfcMmed  via  color  video  tape). 

As  before,  individuals  spending  more  total  time  were  found  to  devote  more  cognitive  time 

per  test,  although  the  effect  was  less  consistent.  These  individuals  also  tended  to  perform  slightly 
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more  tests  than  their  more  effective  counterparts.  Appendix  C  provides  the  data  for  the  three 
studies,  grouped  by  total  fault  diagnosis  time. 

Problem  Effects 

Possibly  the  best  opportunity  for  understanding  more  about  the  ways  system  design  can 
affect  cognitive  workload  is  to  explore  ways  in  which  diagnostic  performance  varied  across 
individual  problems.  A  central  question  concerns  the  source  of  higher  cognitive  time  requirements 
for  some  problems  over  others.  Does  increased  cognitive  workload  result  from  greater  numbers  of 
tests,  higher  cognitive  time  per  test,  or  both? 


I  I 


Figure  4  displays  the  average  cognitive  time  per  problem  for  each  of  the  sixteen  faults 
inserted  into  the  IR  transmitter/receiver  system.  The  problems  arc  presented  in  order  of  increasing 
cognitive  time.  Figure  S  displays  the  average  number  of  tests  and  the  average  cognitive  time  per 
test,  for  each  problem,  shown  in  the  same  order  as  in  Figure  4.  For  this  system,  higher  cognitive 
workload  resulted  from  both  increased  numbers  of  tests  and  increased  cognitive  time  per  test. 
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Figure  5.  Number  of  Tests  and  Cognitive  Time  per  Test,  by  Problem  (IR  System). 

For  the  other  two  systems  studied,  increased  cognitive  time  on  particular  faults  was  a  result 
of  increased  cognitive  time  per  step  only;  the  more  time-consuming  problems  did  not  involve  more 
tests  being  performed.  It  is  important  to  note,  however,  that  the  cognitive  times  per  problem 
varied  only  by  a  factor  of  approximately  two,  for  these  two  systems,  while  the  longest  cognitive 
time  is  ten  times  the  shortest,  for  the  IR  System.  In  general,  the  results  obtained  for  the  IR  system 
are  believed  to  be  the  more  typical,  as  the  experimental  conditions  were  extremely  realistic  for  that 
study. 


SECTION  IIL 

PROJECTING  COGNITIVE  TIME  WORKLOAD  IN  FAULT  DIAGNOSIS 


The  approach  used  by  PROFILE  to  project  manual  times  to  perform  operations  is  one  of 
synthesis;  it  predicts  the  sequence  of  manual  tasks  which  will  be  performed  to  accomplish  some 
goal,  and  then  sums  the  times  of  each  of  the  manual  elements.  The  results  correspond  well  with 
observed  times  because  a  body  of  time  data  exists  for  assessing  the  duration  of  manual  actions. 

A  similar  approach  to  predicting  cognitive  times  would  require  the  ability  to  predict  the 
cognitive  activities  performed,  and  the  times  required  to  accomplish  those  mental  functions.  While 
this  might  be  accomplished  with  some  success  for  extremely  simple  mental  tasks,  such  as  some 
types  of  choice  tasks,  we  have  neither  the  ability  to  predict  the  cognitive  content  of  a  complex 
diagnosis  activity  nor  the  means  for  quantifying  the  time  requirements  of  those  mental  processes. 

The  approach  taken,  therefore,  was  to  attempt  to  find  characteristics  of  the  projected 
diagnostic  sequence  which  account  for  as  much  variation  in  cognitive  time  across  problems  as 
possible.  More  specifically,  three  variables  were  explored  for  the  extent  to  which  they  relate  to 
cognitive  time;  1 )  number  of  tests  performed,  2)  total  manual  time  in  the  diagnosis  sequence,  and  3) 
u  manual  times  of  the  individual  tests  in  the  sequence.  In  all  three  cases,  PROFILE  is  used  to 
predict  the  value  of  the  variable  of  interest 

Appendix  D  presents  a  detailed  listing  of  the  PROFILE  projections,  upon  which  the  cognitive 
time  projections  are  based.  The  projections  are  summarized  in  Appendix  E.  The  projections  of 
cognitive  time  shown  in  these  appendices  are  based  upon  estimates  of  inter-step  cognitive  times,  as 
discussed  later  in  this  section. 

The  annotation  used  in  the  remainder  of  this  section  employs  three  basic  variables;  C  which 
is  actual  cognitive  time,  T  which  is  actual  number  of  tests,  and  M  which  is  the  actual  manual  time. 
Estimates  of  these  are  indicated  as  primed  variables;  fcK*  example  T  and  M'  are  PROFILE- 
generated  estimates  of  the  number  of  tests  required  to  resolve  a  fault  and  the  manual  testing  time 
involved,  respectively.  Values  obtained  as  a  result  of  curve-fitting  are  indicated  with  for 

A 

example  an  estimate  of  C,  using  a  curve  fit  to  experimental  data  is  C. 

Number  of  Tests 

Figures  4  and  5  showed  that,  for  the  IR  System,  actual  cognitive  time  per  problem  increases 
as  actual  number  of  tests  perftvmed  increases.  The  cognitive  times  for  all  thirty  problems  are 
shown  in  Figure  6  versus  the  number  of  tests  in  the  PROFILE  solutions  to  the  problems.  The 


number  of  tests  in  some  PROFILE  solutions  are  non-integers,  since  PROFILE  employs  a  sampling 
process  when  it  must  resort  to  sequential  replacements.  This  technique  yields  a  set  of  reasonable 
solutions  to  those  problems  in  which  multiple  components  are  replaced  in  an  arbitrary  sequence. 


While  the  computed  times  correspond  rather  well  with  the  actual  times,  the  generality  of  the 
function  between  number  of  tests  and  cognitive  time  is  unknown.  If  a  system  design  imposes  tests 
averaging  ten  minutes,  for  example,  the  cognitive  time  per  test  is  expected  to  be  much  greater,  and 
the  projected  cognitive  time,  according  to  the  function  given  above  would  be  low.  If  the  mean  test 
times  for  some  design  are  very  short,  the  opposite  is  true.  For  these  reasons  manual  time  is  now 
considered,  as  an  indicator  of  cognitive  time. 


Manual  Time  Per  Problem 

The  second  analysis  attempts  to  relate  cognitive  time  to  the  total  manual  time  computed  by 
PROFILE  to  diagnose  and  resolve  each  fault.  As  shown  in  Figure  8,  the  relationship  between 
actual  cognitive  time  per  problem  and  projected  manual  dme  is  not  reliable  when  the  problems  from 
all  three  systems  are  pooled  (linear  regression  yields  R=0.46). 

When  the  data  are  analyzed  by  system,  the  times  are  found  to  vary  closely  (and  linearly)  with 
total  manual  time,  but  the  parameters  of  those  functions  are  significantly  different  for  each  of  the 
three  systems.  Cognitive  dme  and  manual  dme  seem  strongly  related,  but  not  in  a  manner  which  is 
general  across  domains.  The  basic  difficulty  is  that  the  cognitive  time  for  a  ten-minute  problem  will 
differ  significantly  if  those  ten  minutes  arose  from  one  or  two  tests,  as  opposed  to  many  tests. 
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Figure  8.  Manual  and  Cognitive  Times,  by  Problem 


The  following  section  investigates  cognitive  times  within  a  fault  diagnosis  sequence,  with  the 
objective  of  developing  a  relationship  which  is  sensitive  to  numbers  of  tests  performed  and  to  the 
manual  workload  of  the  individual  tests  involved. 

Manual  Times  of  Individual  Tests 

The  third  approach  uses  a  form  of  synthesis,  in  which  the  cognitive  time  for  a  problem  is 
computed  as  the  sum  of  the  cognitive  times  projected  to  occur  between  each  two  tests,  i.e.,  the 
interstep  cognitive  times.  The  objective  is  to  derive  a  function  which  is  sensitive  to  the  manner  in 
which  manual  time  was  expended,  i.e.,  whether  it  arises  from  performing  many  brief  tests  or  a  few 
long  tests.  Unfortunately,  we  currently  have  no  way  to  know  what  part  of  an  observed  inter-step 
cognitive  time  was  devoted  to  assessing  the  results  of  the  previous  test,  and  what  part  was  spent 
planning  the  next  action.  Very  specialized  studies  would  be  required  to  obtain  these  time  portions 
individually.  All  efforts  to  discover  a  multi-variate  relationship  between  inter-step  cognitive  time 
and  characteristics  of  both  the  preceding  and  following  test  were  entirely  unsuccessful. 

One  relatively  stable  relationship,  however,  was  obtained  for  the  three  systems  studied 
which  relates  inter-step  cognitive  time  to  the  manual  time  of  the  following  test  Figure  9  presents 
the  average  cognitive  time  preceding  a  test  versus  the  manual  time  of  the  test.  Each  data  point 
shown  represents  the  average  of  all  the  inter-step  cognitive  times  preceding  manual  tests  of  a 
particular  duration,  across  all  three  systems  studied  (Appendix  F  presents  the  data  values;  the  data 
were  accumulated  for  each  diagnostic  episode  until  the  first  replacement  was  made).  The  fust  data 
point  shown,  for  example,  indicates  that  the  mean  cognitive  time  preceding  a  manual  test  of  five 
seconds  in  duration  was  9.7  seconds  (N=86). 


The  curve  fit  to  these  data  is 


IC=  10  +  M’0  6S 

A 

where  IC  is  the  projected  inter-step  cognitive  time 

M'  is  the  manual  time  of  the  following  test,  as  projected  by  PROFILE 

This  relationship  appears  most  promising  as  a  means  for  assessing  cognitive  workload,  for  it 
appears  to  be  generally  correct  for  each  of  the  individual  systems  studied  as  well.  The  only 
difference  found  when  interstep  cognitive  times  were  computed  by  system  was  the  Y-intercept, 
representing  a  constant  per  test,  which  was  14  seconds  for  the  IR  system,  and  8  seconds  for  the 
other  two  systems  combined.  More  importantly,  the  exponent  of  0.65  provided  a  good  fit  to  the 
results  summarized  by  individual  system. 

Appendix  D  lists  the  PROFILE  diagnosis  sequences  for  each  of  the  thirty  faults  studied. 

This  listing  also  displays  the  inter-step  cognitive  dmes  computed  according  to  the  function 
10 +  M* 


Computing  Total  Cognitive  Times 


A 

The  total  estimated  cognitive  time  to  diagnose  a  fault  is  simply  the  sum  of  the  IC  computed  at 


each  test;  C  =  Z  IC. 


Figure  10  displays  the  total  projected  cognidve  times  for  the  thirty  problems  in  the  three 
systems  plotted  against  the  actual  average  cognitive  time  per  problem,  across  all  subjects.  As  seen 
in  this  figure,  the  PROFILE  projections  tend  to  be  significantly  low  for  three  problems  involving 
high  actual  cognitive  time. 
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Figure  10.  Projected  Cognitive  Times  per  Problem,  based  on  PROFILE 
Projections  of  Inter-step  Cognitive  Time 
15 


These  faults  (IR  faults  1, 6,  and  9),  are  also  ones  which  required  significantly  more  steps  to 
solution  by  PROFILE.  When  number  of  PROFILE  steps  is  included  in  the  projection  function,  the 
following  achieves  the  best  fit  to  the  data: 

C  =  ZIC  -59  +  r2‘‘ 

A 

where  C  is  the  projected  cognitive  time  to  diagnose  a  fault 

IC  is  the  inter-step  cognitive  time  prior  to  each  PROFILE  test,  computed  as  10  + 

T  is  the  number  of  PROFILE  tests 

Figure  1 1  displays  the  projected  cognitive  times  for  the  thirty  problems  using  this  function, 
versus  the  actual  cognitive  times  (R  =  0.755;  F=37.082). 
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Figure  12.  Projected  and  Actual  Total  Maintenance  Times 

As  seen  from  Figures  7,  10,  and  12,  one  particular  fault  caused  the  technicians  considerably 
more  difficulty  than  would  be  predicted  by  the  characteristics  of  PROFILE  diagnosis  sequences. 
This  fault  is  problem  six,  IC  46,  a  2-input  AND  gate  failed  in  the  IR  System,  as  shown  in  the 
schematic  diagram  in  Appendix  G.  This  component  happens  to  be  one  of  several  in  a  feedback 
circuit.  When  it  fails,  the  circuit  runs  continuously,  rather  than  halting  periodically  to  reset 
counters.  To  the  troubleshooter,  the  observable  signals  may  appear  generally  normal.  If  the 
abnormality  is  detected,  the  symptom  does  not  assist  in  identifying  the  faulty  component.  Since  the 
output  of  IC  46  is  an  extremely  brief  reset  pulse,  there  is  no  effective  way  to  detect  it  with  standard 
test  equipment.  Thus,  in  this  design  the  troubleshooter  must  ultimately  resort  to  sequential 
replacement  until  the  fault  is  resolved. 

The  functions  developed  here  to  reflect  cognitive  time  workload  only  parually  recognize  such 
technical  difficulties  posed  by  the  system  design.  While  the  existence  of  a  feedback  loop  was  not 
explicitly  specified  to  PROFILE,  the  troubleshooting  model  did  encounter  the  same  difficulty  in 
isolating  the  fault  as  the  actual  technicians  did,  i.e.,  it  resorted  to  sequential  replacement  when  no 
effective  test  could  be  found  to  discriminate  the  true  source  of  the  abnormality.  The  fact  that  the 
panicular  signal  being  tested  was  difficult  to  interpret,  however,  was  not  considered  in  the  function 
for  cognitive  time. 

Ultimately,  it  may  be  possible  to  develop  a  more  comprehensive  projection  technique  which 
is  sensitive  to  a  wider  range  of  conditions.  The  following  section  addresses  this  objective. 


SECTION  IV.  COGNITIVE  COMPLEXITY  IN  FAULT  DIAGNOSIS 


The  variables  used  in  the  previous  section  to  project  cognitive  time  are  indirect  measures  of 
cognitive  workload.  Tlif'y  vary  with  actual  cognitive  time  variations  because  they  reflect  the 
difficulty  PROFILE  has  in  diagnosing  a  problem.  Within  a  particular  system  design,  faults  which 
require  more  PROfTLE  tests  or  lengthier  manual  testing  are  faults  which  either  do  not  produce 
many  abnormal  symptoms,  and  are  therefore  difficult  to  detect,  or  whose  symptoms  are  difficult  to 
discriminate  from  those  of  other  faults,  or  both.  This  is  true  because  the  PROFILE  model  attempts 
to  resolve  problems  in  the  most  effective  manner  possible,  and  resorts  to  lengthy  or  numerous  tests 
only  when  forced  to  by  the  system  design  and  the  nature  of  the  fault. 

The  projections  of  cognitive  time  workload  should  stand  up  well  across  electronic  systems, 
for  the  function  relating  inter-step  cognitive  time  to  manual  time  of  the  following  test  appears  to  be 
relatively  stable  in  somewhat  different  electronic  environments.  There  is  no  evidence,  however, 
concerning  its  applicability  in  domains  where  the  options,  penalties,  and  distributions  of  task  times 
are  widely  different  from  those  encountered  in  electronic  maintenance. 

The  following  sections  discuss  an  analysis  of  cognitive  workload  which  may  allow  assessing 
cognitive  workload  in  a  wider  range  of  domains  in  the  future. 

Types  of  Cognitive  Complexity  in  Fault  Diagnosis 

The  sources  of  cognitive  difficulty  in  diagnosis  arc  numerous.  It  is  useful,  therefore,  to 
somewhat  partition  the  possibilities.  Cognitive  workload  is  viewed  as  being  affected  by  three  main 
influences;  1)  the  complexity  of  the  system  architecture  concerned  with  fulfilling  the  missions  of  the 
system,  termed  operational  complexity,  2)  the  complexity  of  the  process  required  to  diagnose 
faults  in  the  system,  called  process  complexity,  and  3)  the  complexity  of  the  resource-allocation 
and  planning  environment,  termed  executive  complexity. 

These  three  types  of  characteristics  mutually  influence  many  of  same  specific  diagnostic 
functions.  For  example,  we  propose  that  cognitive  time  devoted  to  selecting  a  test  is  affected  by  the 
complexity  of  the  particular  diagnostic  sequence  in  progress,  by  the  complexity  of  the  system  under 
repair,  and  by  the  issues  confronting  the  troubleshooter  at  the  planning  and  resource  allocation 
level.  How  these  factors  combine  to  impact  cognitive  workload  is  not  at  all  clear. 

Operational  Complexity 

Operational  complexity  relates  to  the  ease  of  understanding  the  functions  of  the  system  under 


repair,  in  both  normal  and  failed  conditions.  This  should  not  be  confused  with  such  measures  as 
testability,  which  reflect  the  ease  with  which  system  functions  can  be  monitored.  Operational 
complexity  is  a  characteristic  of  the  way  the  system  is  designed  to  accomplish  its  mission  functions, 
and  it  excludes  consideration  of  design  features  intended  to  facilitate  maintenance  (such  features  are 
considered  in  the  next  section).  While  operational  complexity  can  heavily  influence  ease  of 
maintenance,  the  designer  may  have  relatively  little  opportunity  to  simplify  the  hardware  which 
accomplishes  the  primary  missions  of  the  system.  High  operational  complexity  may  increase  the 
difficulty  of  devising  and  interpreting  tests  which  assist  in  isolating  faults. 

While  operational  complexity  is  affected  by  the  number  of  physical  components  in  the 
system,  the  cognitive  difficulty  of  working  in  the  domain  of  a  particular  design  may  be  much  more 
related  to  the  organization  of  the  components,  and  the  nature  of  the  functions  which  those 
components  perform. 

Currently,  it  is  very  difficult  to  compare  the  complexity  of  the  internal  architecture  of  two 
system  designs.  This  is  largely  so  because  1)  the  forms  of  representing  those  designs  typically 
contain  immense  amounts  of  implicit  information,  and  2)  the  elements  and  interconnections  shown 
may  be  of  differing  degrees  of  internal  complexity.  A  block  diagram  of  one  system,  for  example, 
could  scarcely  be  compared  to  that  of  another  design,  for  the  degree  of  detail  represented  within  the 
blocks  is  a  matter  of  arbitrary  choice.  Even  two  detailed  schematic  diagrams  would  be  exceedingly 
difficult  to  compare,  for  the  components  and  signals  could  be  of  vastly  different  levels  of 
complexity.  Furthermore,  an  assessment  of  cognitive  workload  would  be  very  difficult  to  develop 
based  upon  a  schematic  diagram,  for  these  involve  a  wide  inventory  of  specific  components. 

A  more  feasible  approach  may  be  to  develop  a  more  generalized  form  of  representation  which 
involves  highly  common,  generic  functions.  In  such  a  representation  all  outputs  and  inputs  would 
either  be  of  comparable  complexity,  or  they  would  cany  some  notation  expressing  the  extent  to 
which  they  are  formed  by  combining  simpler  signals.  The  problem  here  is  not  one  of  developing 
an  exotic  representation  form;  a  block  diagram  approach  may  be  adequate.  The  problem  is  in 
representing  a  system  in  a  consistent  form  such  that  measures  could  be  computed  which  reflect  the 
complexity  of  the  system's  operation.  Experimentation  could  then  be  devoted  to  measuring  how 
cognitive  workload  varies  with  this  computed  value. 

The  function  of  some  idealized  systems  currently  can  be  inferred  from  block-diagram 
representations  if  the  following  are  true; 

1.  all  outputs  are  time-invariant,  and  single-valued 

2.  a  failure  in  any  block  causes  all  of  its  outputs  to  be  abnormal 

3.  any  abnormal  input  to  a  block  causes  all  of  its  outputs  to  be  abnormal 


Much  of  the  research  of  troubleshooting  behavior,  including  our  study  two,  has  used 
systems  which  conform  to  these  conditions.  These  conditions  are  rarely  true  for  real-world 
systems,  however,  although  portions  of  some  analog  systems  approach  this  level  of  simplicity. 
More  typically,  some  output  lines  can  carry  a  multitude  of  possible  values  depending  upon  the 
configuration  of  the  system,  failures  in  some  blocks  only  impair  some  of  their  outputs,  and  some 
abnormal  inputs  impact  only  some  of  the  outputs  of  the  blocks  they  drive.  In  the  most  complicated 
cases,  outputs  vary  over  time,  outputs  become  inputs  to  feedback  loops,  and  outputs  are  complex 
combinations  of  multiple  signals. 
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Representations  of  complex  systems  can  often  be  developed  which  meet  conditions  two  and 
three  above,  by  successively  decomposing  complex  system  elements  into  sub-structures.  Of 
course  this  decomposition  yields  a  representation  involving  many  more  blocks  and  explicit 
interconnections  than  were  evident  in  the  original  block  diagram  or  schematic  diagram,  but  in  this 
form  the  system  behavior  is  fully  specified  in  terms  of  comparable  generic  functions  and  signals. 
Other  types  of  system  complexity  resist  simplification  or  representation  in  a  standardized  manner. 

If  a  system  design  requires  complex  signal  paths  and  complex  signal  forms,  there  is  a  limited 
opportunity  for  representing  this  in  more  elementary  forms. 

If  a  standardized  representational  format  can  be  attained,  then  systems  of  widely  differing 
architecture  could  be  compared  by  computing  the  numbers  of  signals  and  signal  conversions,  as 
suggested  by  Leuba  (1962)  or  Wohl  (1980).  Moreover,  experimentation  could  be  undertaken  to 
explore  how  cognitive  workload  varies  in  relation  to  computed  complexity  values. 

It  is  important  to  reiterate  that  operational  complexity  is  expected  to  ir^uence  diagnostic 
workload,  but  that  it  does  hot  fully  determine  the  diagnostic  effort.  Since,  by  definition, 
operational  complexity  addresses  the  technical  environment  in  which  diagnosis  occurs,  it  should  be 
combined  in  some  manner  with  the  implications  of  diagnostic  difficulty  related  to  the  particular 
diagnostic  sequence.  A  technique  for  assessing  the  complexity  of  the  diagnostic  process  is 
presented  below. 

Process  Complexity 

Process  complexity  relates  to  the  cognitive  woildoad  encountered  in  conducting  diagnostic 
performance.  More  specifically,  it  pertains  to  the  functions  of  maintaining  the  information  gained 
by  previous  tests,  in  a  particular  fault  diagnosis  sequence,  and  of  selecting  funire  actions  which  will 
increase  that  information  in  manageable  ways.  Unlike  operational  complexity,  which  reflects  the 
general  technical  environment  in  which  all  fault  diagnosis  in  that  system  occurs,  process  complexity 
is  a  direct  function  of  the  particular  actions  performed,  and  hence  of  the  fault  being  isolated.  Thus 
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some  faults  do  not  require  diagnostic  sequences  which  involve  significant  complexity,  while  others 
force  the  troubleshooter  to  delve  deeply  into  the  more  difficult  sections  of  the  system. 

In  general,  the  system  designer  can  reduce  process  complexity  in  two  ways,  1)  provide  tests 
which  reflect  the  operation  of  system  functions  in  convenient  and  powerful  ways,  and  2)  reduce  the 
requirement  for  the  maintainer  to  develop  ad-hoc  diagnostic  procedures.  Thus  a  system  with  high 
operational  complexity  could  be  made  to  exhibit  low  process  complexity  by  providing  built-in  tests 
which  are  easy  to  conduct  and  interpret,  or  highly  proceduralized  diagnostic  approaches.  When 
these  resources  fail  to  identify  a  fault,  however,  the  maintainer  of  such  a  system  must  then  generate 
a  testing  sequence  in  an  environment  of  high  operational  complexity 

The  difficulty  of  attaining  a  cognitive  objective,  like  identifying  the  source  of  a  fault,  is 
determined  by  the  particular  method  employed  to  attain  the  objective.  When  measuring  the  manual 
difficulty  of  attaining  some  objective,  like  tuning  an  automobile  engine,  we  must  first  specify  how 
the  work  is  to  be  done.  Inexperienced  mechanics  may  pursue  methods  which  cause  the  work  to  be 
very  difficult,  while  more  experienced  individuals  may  be  able  to  avoid  some  of  the  difficulty. 

Thus  manual  difficulty  is  not  a  quality  which  is  inherent  to  an  objective  but  is  inherent  to  the  means 
for  attaining  the  objective.  Similarly,  cognitive  difficulty  can  only  be  defined  in  terms  of  a 
particular  sequence  of  cognitive  activities  performed. 

As  with  manual  performance,  there  are  great  variations  among  individuals  in  the  cognitive 
activities  they  perform  to  attain  identical  objectives.  Which  of  all  these  possible  approaches,  then, 
should  be  used  as  the  basis  for  evaluating  the  cognitive  difficulty  of  a  particular  task?  One  feasible 
approach  is  to  base  this  evaluation  upon  the  decisions  made  by  an  expert  troubleshooter.  Thus  if 
we  know  that  an  expert  will  resolve  a  particular  fault  in  a  particular  manner,  we  can  retrospectively 
analyze  the  character  of  the  task  at  each  stage.  Since  we  have  a  model  of  an  expert  troubleshooter 
in  PROFILE,  we  can  use  its  diagnostic  strategies  as  the  source  of  this  evaluation. 

The  complexity  at  each  stage  of  a  testing  sequence  is  determined  by  evaluating  the  suspicions 
which  exist  as  a  result  of  all  previous  tests.  If,  for  example,  the  previous  tests  in  a  sequence 
indicate  that  the  fault  must  be  in  Unit  A  of  a  system,  and  cannot  be  in  Unit  B,  then  the  suspicion 
state  at  this  stage  is  extremely  simple.  Furthermore,  the  complexity  at  this  stage  is  considered  to  be 
independent  of  the  internal  complexity  of  each  of  the  units.  In  succeeding  tests  the  complexity  of 
the  separate  units  may  have  to  be  confronted,  but  at  this  particular  stage  it  can  be  avoided  by  the 
diagnostician. 

If,  on  the  other  hand,  previous  test  results  point  to  certain  sub-sections  within  Unit  A,  and 
some  parts  of  Unit  B,  then  the  suspicion  state  is  more  complicated. 
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Example.  Suppose  the  system  under  consideration  is  represented  as  shown  in  Figure  13.  In 
this  example  there  are  two  modules  (I  and  II),  five  boards  (A,B,C,D,E),  and  twelve  Replaceable 
Units  (RU's). 


Module  I 


Module  n 


Board  A 
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Board  C 


Board  D 


Board  E 


Figure  13.  A  Sample  System  Hierarchy 

At  any  stage  of  diagnosis  we  may  attach  a  probability  to  each  RU,  representing  the 
relative  suspicion  we  hold  that  the  fault  is  in  that  RU.  Initially,  these  probabilities  are  computed 
from  our  estimates  of  the  relative  failure  rates  of  the  RU’s.  As  testing  progresses  we  can  revise  the 
probabilities  according  to  the  evidence  which  each  test  produces.  The  exact  value  of  these 
suspicion  levels  is  relatively  unimportant.  For  the  purpose  of  identifying  what  sections  of  a  system 
are  under  significant  suspicion  it  is  probably  sufficient  to  convert  all  suspicion  levels  into  High'  or 
'Low'.  If  the  system  design  provides  tests  which  map  cleanly  onto  the  system  organization,  then 
the  high  and  low  suspicion  levels  will  fall  into  groups  matching  units  in  the  system  representation. 

Figure  14  displays  sample  probabilities  which  might  exist  at  some  step  of  diagnosis.  The 
probabilities  of  higher  levels  of  organization  are  simply  the  sums  of  their  sub-element  probabilities. 
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Figure  14.  Current  Suspicion  Levels 


From  Figure  14,  it  is  seen  that  all  the  components  on  boards  D  and  E  are  suspected  to  a 
meaningful  degree,  as  is  one  component  in  board  A.  A  verbal  characterization  of  the  current 
suspicion  state  is  "we  suspect  all  components  in  module  II,  and  component  1  in  board  A  of  module 
I."  This  characterization  is  made  as  concise  as  possible  by  grouping  all  significant  suspicions  as 
much  as  possible. 


Figure  15  displays  this  characterization  graphically.  Note  that  sub-groups  of  system 
elements  are  combined  into  higher  levels  whenever  their  suspicion  levels  arc  similar.  If  some 
elements  are  suspected  highly  and  others  arc  not,  then  no  further  grouping  is  done. 
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Figure  IS.  Suspicion  Characterization 
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Now  consider  an  alternative  state  of  suspicions,  as  shown  in  Figure  16.  This  state  might 
result  from  performing  a  different  test  than  that  performed  to  produce  Figure  14.  Here,  all 
components  in  module  11  arc  suspected  highly,  and  no  component  in  module  I  is  suspected  highly 
This  state  could  be  the  result  of  a  single  test  which  monitors  the  entire  functioning  of  one  of  the 
modules  and  is  in  no  way  affected  by  the  other,  or  it  could  reflect  the  cumulative  effects  of  a  series 
of  tests. 
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Figure  16.  Suspicion  Levels  Resulting  From  an  Alternative  Test 


The  suspicions  shown  in  Figure  16  may  be  stated  as  "we  suspect  everything  in  module  n 
and  nothing  in  module  I."  Figure  17  illustrates  the  suspicion  situation  graphically,  after 
simpliHcation. 
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Figure  17.  Simplified  Suspicion  State  Following  Alternative  Test 


Quantifying  Complexity  at  a  Stage.  The  following  formula  is  used  to  quantify  the 
complexity,  C  ,  of  a  suspicion  state,  in  terms  of  the  suspicion  levels  of  the  sub-elements  at  a  stage 
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in  a  diagnostic  sequence: 
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Cp=  -Z  Pi  In  Pi 

where  Cp  is  the  complexity  of  the  suspicion  state  at  the  stage  in  the  sequence 
Pi  is  the  probability,  or  suspicion  level,  of  the  ith  element  at  that  stage 

This  classical  expression  for  entropy  yields  higher  values  when  many  elements  are  suspected 
equally,  and  lower  values  when  there  are  a  few  elements  or  when  most  of  the  uncertainty 
(suspicion)  is  centered  upon  a  few  elements.  According  to  this  expression,  the  complexity  of  the 
sample  system  shown  in  Figure  15  is  as  follows: 

Cp  =  -  (.15  In  .15  +  .02  In  .02  +.02  In  .02  +  .02  In  .02  +  .04  In  .04  +  .75  In  .75) 

=  -(  -.28  -.08  -.08  -.08  -.13  -.22) 

=  -  (-0.87) 

=  0.87 

whereas  the  complexity  of  the  suspicion  state  shown  in  Figure  17  is 
Cp  =-(.14 In.  14 +  .86 In  .86) 

=  -(-.28 -.13) 

=  0.41 

In  summary,  the  complexity  of  a  suspicion  state  is  determined  by  the  following  process: 

1 .  determine  a  suspicion  level  for  each  replaceable  unit  (RU)  in  the  system 

2.  recursively  combine  all  elements  which  are  suspected  to  a  similar  degree 

3.  compute  the  complexity  of  the  resulting  state,  as  Cp  =  -Z  pj  In  pj 

In  general,  the  state  complexity  will  be  low  at  the  beginning  of  a  diagnostic  sequence,  for 
early  tests  tend  to  address  high-level  units  of  system  hardware,  and  therefore  low-level 
subcomponents  are  unlikely  to  be  suspected  to  widely  differing  degrees.  As  testing  proceeds, 
however,  more  specialized  tests  are  performed,  which  monitor  some  of  the  low-level  functions. 
When  this  occurs  the  suspicion  state  is  likely  to  become  far  more  complex.  Late  in  a  diagnostic 
sequence,  the  state  complexity  will  usually  return  to  a  low  level,  for  the  suspicions  are  generally 
confined  to  a  few  low-level  elements.  If  even  a  few  such  suspected  elements  reside  in  different 
system  functions  or  hardware  units,  however,  the  complexity  can  be  high  until  the  fault  is  finally 
identified. 
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Compiitin?  Total  Process  Complexity.  The  total  process  complexity  for  a  diagnosis 
sequence  is  computed  as  the  sum  of  the  state  complexity  values,  Cp,  following  each  test.  Unlike 
operational  complexity,  the  measure  of  process  complexity  is  sensitive  to  the  symptoms  produced 
by  a  particular  fault,  and  to  the  maintainability  features  of  the  system  design.  If  a  particular  fault 
can  be  isolated  easily,  and  with  few  tests,  then  the  total  process  complexity  will  be  low.  Faults 
which  force  the  troubleshooter  to  perform  many  tests,  or  tests  which  do  not  map  well  onto  the 
current  suspicion  state,  will  yield  higher  values. 

The  measure  of  process  complexity  offers  several  advantages  over  more  classical  approaches 
to  assessing  the  complexity  of  a  system  design,  such  as  counting  the  number  of  signals  and  system 
elements.  First,  it  is  sensitive  to  the  relationship  between  the  system  design  and  the  maintainability 
features  provided  to  monitor  system  functions.  A  system  can  be  composed  of  an  immense  number 
of  components,  yet  reflect  ease  of  maintenance  in  the  process  complexity  measure,  if  the  tests 
available  to  the  maintainer  allow  for  easy  partitioning  of  suspicion  about  the  source  of  the  fault. 
Some  computers,  for  example,  are  constructed  of  a  very  high  number  of  components,  yet  are 
relatively  easy  to  maintain  because  the  tests  are  easily  performed  and  interpreted.  A  second 
advantage  of  the  process  complexity  measure  is  that  it  can  identify  particular  faults  which  involve 
relatively  high  diagnostic  complexity.  This  affords  the  designer  an  opportunity  to  explore  design 
revisions  which  could  relieve  the  difficulties.  Finally,  a  potential  benefit  of  computing  total  process 
complexity  from  individual  state  complexities  is  that  it  may  be  found  that  cognitive  workload  is 
related  more  to  the  maximum  complexity  encountered  within  a  problem  than  to  the  sum  of  the  state 
values.  Since  all  suspicion  state  values  are  available  within  PROFILE,  it  will  be  possible  to  retain 
both  the  maximum  complexity  level  and  the  length  of  time  the  expert  troubleshooter  remained  in 
each  complexity  state. 

Two  additional  comments  should  be  made  about  the  procedure  for  computing  process 
complexity.  First,  there  are  almost  always  alternative  hierarchical  representations  of  a  system.  A 
representation  could  reflect  the  physical  packaging  of  the  system,  its  functional  relationships,  or 
some  hybrid  combination  of  the  two.  Even  when  focusing  on  just  one  type,  such  as  the  functional 
hierarchy,  two  experts  can  differ  in  representing  the  system.  The  implication  of  this  is  that  process 
complexity  depends  upon  the  representation  of  the  system.  If  one  individual  views  the  system 
physically,  as  novices  tend  to  do,  then  the  complexity  of  that  person's  diagnosis  strategy  will  be 
deeply  affected  by  this  conception.  If  experts  conceive  of  a  system  in  more  functional  terms,  then 
the  complexity  follows  this  form.  Most  likely,  experts  have  alternative  representations  which  they 
employ  depending  upon  the  circumstances.  Similarly,  some  system  designs  may  encourage 
functional  thinking,  while  others  may  be  easily  viewed  with  physical  representations. 

Secondly,  it  is  recognized  that  the  exact  procedure  for  combining  like-suspected  elements  has 
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not  been  specified  in  complete  detail  here.  In  the  examples,  suspicions  were  considered  liigh'  if 
the  level  exceeded  about  0.10,  and  'low'  otherwise.  Many  other  absolute  and  relative  schemes  are 
possible  for  combining  elements  into  larger  conceptual  chunks,  and  some  of  these  have  been 
informally  explored.  Until  more  specialized  studies  are  conducted,  however,  the  best  way  to 
perform  this  step  will  remain  unclear.  The  objective  ought  to  be  to  detennine  what  scheme 
produces  suspicion  representations  most  like  those  held  by  actual  maintainers. 

Executive  Complexity 

Executive  complexity  relates  to  the  significance  of  the  options  which  must  be  weighed  by  the 
troubleshooter  in  managing  fault  diagnosis  performance.  In  general,  the  time  to  decide  upon  a 
particular  action  will  be  affected  by  the  gravity  of  the  actions  under  consideration.  The  data  support 
the  notion  that  when  the  maintainer  is  about  to  perform  a  lengthy  action,  or  one  involving 
replacement  of  more  expensive  units  of  hardware,  the  cognitive  decision  time  increases.  This  effect 
may  also  be  true  when  the  impending  action  is  either  dangerous,  uncomfortable,  or  error-prone. 

In  some  corrective  maintenance  settings  the  implications  of  alternatives  is  quite  high.  One 
test  might  involve  a  time-consuming  and  costly  configuration  of  people  and  equipments,  while 
another  could  involve  an  error-prone  or  dangerous  disassembly  which  could  create  a  need  to 
perform  a  difficult  re-calibration.  In  such  a  case  the  troubleshooter  must  weigh  alternatives  with 
significant  implications,  and  would  be  expected  to  expend  considerably  more  time  considering  the 
issues. 

The  interstep-cognitive  time  data  described  in  section  m  provided  a  function  relating 
cognitive  time  to  the  manual  performance  time  of  the  impending  action.  What  those  data  could  not 
indicate  was  what  portion  of  the  time  function  was  a  result  of  the  operational  complexity  of  the 
particular  system  under  test,  what  part  was  related  to  the  complexity  of  the  particular  diagnostic 
process,  and  what  part  was  influenced  by  the  available  choices.  One  hypothesis  is  that  the  intercept 
of  the  function  is  related  in  some  manner  to  the  operational  complexity  of  the  system  under  repair, 
the  exponential  form  of  the  curve  is  related  to  the  time  significance  of  the  alternatives,  and  the  large 
variations  about  the  mean  time  values  for  each  sample  point  were  due  to  variations  in  the  process 
complexity  which  were  not  controlled. 

Currently,  PROFILE  can  detect  decision  points  involving  significant  time  implications,  and  it 
can  recognize  when  a  replacement  of  a  costly  spare  part  is  about  to  be  performed.  It  could  also 
incorporate  considerations  of  test  danger  or  discomfort  into  its  decision  process,  if  test  cost, 
currently  expressed  in  time,  were  weighted  by  an  appropriate  factor  reflecting  the  environment  in 
which  the  test  is  performed.  PROFILE  cannot  currently  deal  with  the  very  real  issue  of  human 
error,  however. 
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SECTION  V.  SUMMARY  AND  CONCLUSIONS 


Fault  diagnosis  is  regarded  primarily  as  a  cognitive  activity  supported  by  the  performance  of 
manual  actions  which  are  performed  to  obtain  new  information.  This  does  not  imply  that  the 
majority  of  time  diagnosing  a  fault  is  purely  cognitive.  In  three  experimental  studies  conducted  of 
fault  diagnosis  in  electronic  maintenance,  the  average  time  devoted  to  purely  cognitive  activity 
ranged  from  approximately  one-third  to  one-half  the  total  corrective  maintenance  time. 

Summary 

The  central  objective  of  this  investigation  was  to  develop  a  generalized  procedure  for 
projecting  cognitive  time  expended  during  fault  diagnosis.  The  approach  was  to  search  for 
characteristics  of  experimentally  observed  diagnostic  performance  which  were  related  to  actual 
cognitive  time  expenditures,  and  then  to  project  those  characteristics  and  associated  times  for  a 
particular  system,  and  sample  of  faults,  using  PROFILE.  Toward  this  end,  a  total  of  638  fault 
diagnosis  episodes  were  analyzed  in  detail.  Guided  by  the  findings  of  this  analysis,  three 
alternative  approaches  to  projecting  cognitive  time  were  derived,  one  of  which  appears  to  be 
relatively  reliable  across  system  designs.  This  approach,  which  is  a  function  of  the  manual  times  of 
the  individual  tests  and  the  number  of  tests  performed,  has  been  implemented  within  PROFILE  for 
use  in  projecting  total  corrective  maintenance  times. 

A  fundamentally  different,  and  yet  more  ambitious,  approach  to  projecting  cognitive 
workload  is  to  attempt  to  identify  generic  sources  of  cognitive  difficulty  which  can  be  quantitatively 
evaluated  for  particular  faults  in  particular  system  designs.  This  report  has  attempted  to  outline 
what  those  generic  types  of  complexity  might  be,  and  how  they  may  be  quantified  individually. 

Experimentally  Observed  Diagnostic  Performance 

In  all  three  studies  of  diagnostic  performance,  cognitive  time  per  fault  diagnosed  was 
significantly  correlated  to  all  of  the  following;  1)  number  of  tests  performed,  2)  manual  time 
expended  in  testing  and  3)  cognitive  time  per  test. 

An  analysis  of  individual  diagnostic  performance  showed  that  individuals  who  expended 
more  cognitive  time  in  fault  diagnosis  did  not  differ  significantly  from  those  who  expended  less 
cognitive  time,  in  terms  of  the  manual  time  spent  testing  and  the  number  of  tests  performed. 
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Individuals  who  completed  fault  diagnosis  in  the  least  total  time  (manual  plus  cognitive), 
however,  generally  expended  less  cognitive  time  and  less  manual  time,  they  performed  fewer  tests, 
and  they  expended  less  cognitive  time  per  test,  than  diagnosticians  requiring  more  total  time. 

Faults  which  required  increased  cognitive  time  to  solution  generally  required  more  tests,  and 
higher  cognitive  time  per  test,  than  faults  involving  a  lower  cognitive  time  workload. 
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Projecting  Cognitive  Time  Workload 

The  most  pronusing  characteristics  of  PROFILE  diagnostic  sequences,  for  the  purpose  of 
projecting  cognitive  lime,  were  found  to  be  1 )  the  manual  times  of  the  individual  tests  performed, 
and  2)  the  number  of  tests  to  diagnose  a  fault.  When  cognitive  time  is  computed  as  a  function  of 
these  two  factors,  the  projections  of  cognitive  time  are  relatively  accurate. 

Generic  Complexity  Types  in  Fault  Diagnosis 


Three  types  of  complexity  are  defined.  These  arc  considered  to  mutually  influence  the 
cognitive  workload  in  fault  diagnosis. 

Operational  complexity  reflects  the  complexity  of  the  system  under  repair,  in  general. 
Quantification  of  this  characteristic  is  feasible  in  the  relatively  near  future,  if  means  can  be 
developed  for  expressing  system  designs  in  terms  of  comparable  simple  functions  and  simple 
outputs. 

Process  complexity  reflects  the  nature  of  the  diagnostic  sequence  performed  by  an  expert 
troubleshooter  to  identify  a  particular  fault,  and  is  computed  by  summing  the  complexity  of  the 
suspicion  state  at  each  stage  of  fault  isolation.  The  suspicion  state  is  formed  by  manipulating  the 
system  hierarchy  such  that  it  represents  the  major  system  elements  suspected  and  not  suspected, 
following  each  test  This  technique  identifies  faults  which  involve  complicated  testing  sequences 
and  it  is  sensitive  to  aspects  of  the  system  design  which  complicate  diagnosis. 

Executive  complexity  has  to  do  with  the  significance  of  the  choices  facing  the  troubleshooter. 
If  the  alternatives  under  consideration  involve  serious  time,  cost,  or  personal  safety  factors,  then 
executive  complexity  increases,  and,  apparently  so  does  the  cognitive  time  involved  in  the  decision. 
The  data  analyzed  in  this  research  provided  one  view  of  executive  complexity,  in  the  form  of 
varying  time  costs  of  tests.  The  relationship  between  actual  inter-step  cognitive  times  and  the 
manual  times  of  the  following  tests  indicates  that  cognitive  time  increases  as  the  time  of  the  test 
under  consideration  increases,  but  rapidly  approaches  an  asymptote. 
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Conclusions 


This  research  represents  a  beginning  in  attempting  to  quantify  cognitive  workload  in  fault 
diagnosis.  The  retrospective  analysis  of  experimentally  obtained  performance  data  reveals 
significant  relationships,  suggesting  that  expenditure  of  cognitive  time  is  related  to  specific 
characteristics  of  the  diagnostic  context  and  content.  The  PROFILE  model  of  expert 
troubleshooting  strategy  has  provided  a  resource  for  quantifying  those  characteristics,  and,  with 
some  adjusting  for  internal  bias,  has  produced  diagnostic  routines  whose  characteristics  correspond 
in  meaningful  ways  with  actual  cognitive  times. 

Even  very  strong  relationships  among  variables,  however,  do  not  satisfy  the  requirements 
for  a  projection  technique  until  and  unless  the  parameters  of  those  relationship  are  validated  for 
generality  across  a  number  of  domains.  The  relationship  between  inter-step  cognitive  time  and  the 
manual  time  of  the  following  test  was  examined  in  three  relatively  diverse  experimental  settings, 
and  found  to  be  quite  similar  across  those  domains.  Further  experimentation  should  be  conducted 
1)  to  test  this  relationship  across  a  wider  range  of  diagnostic  environments,  2)  to  explore  its 
generality  at  much  higher  time  values,  and  3)  to  determine  what  aspects  of  the  relationship  are 
affected  by  the  design  of  the  system,  and  the  particular  complexity  of  the  diagnostic  process  at  the 
time  of  performance. 

This  work  was  greatly  facilitated  by  an  unusually  substantial  body  of  actual  troubleshooting 
performance  data.  Because  of  the  relatively  realistic  environment  in  which  these  studies  were 
conducted  (i.e.,  the  troubleshooters  were  not  asked  to  explain  their  work,  nor  were  they  interrupted 
in  any  manner  to  facilitate  data  collection),  the  validity  of  the  data  is  believed  to  be  very  high. 

As  would  be  expected,  this  experimental  approach  also  had  associated  disadvantages.  Since 
the  original  studies  of  diagnostic  performance  were  not  conducted  for  the  primary  purpose  of 
exploring  cognitive  workload,  a  number  of  critical  questions  could  not  be  examined.  Interestingly, 
however,  there  are  a  number  of  examinations  of  the  existing  data  which  could  yield  more 
understanding,  but  remain  to  be  performed.  The  most  promising  of  these  would  be  a  study  of  the 
variations  of  inter-step  cognitive  times  prior  to  tests  of  approximately  the  same  time  duration.  It  is 
suspected  that  these  variations  could  be  largely  explained  in  terms  of  the  particular  process 
complexity  that  existed  at  each  step. 

The  types  of  studies  which  should  now  be  conducted  to  resolve  many  of  the  questions  raised 
here  are  now  much  clearer  as  well.  Of  particular  value  would  be  the  observation  of  a  large  number 
of  carefully  controlled  decision  tasks,  including  the  following: 

Cognitive  time  related  to  symptom  assessment.  The  measure  of  interest  is  the  time  required  to 


judge  the  normality  of  a  test  result.  Within  a  few  systems,  subjects  would  be  given  a  sequence  of 
test  symptoms,  and  be  asked  to  judge  the  normality  of  each  reading. 


Cognitive  time  related  to  symptom  interpretation.  The  measure  of  interest  is  the  time  required  to 
interpret  a  test  result  in  terms  of  the  possible  sources.  Subjects  would  be  given  test  results, 
including  whether  those  results  are  normal  or  abnormal,  and  be  asked  to  interpret  the  significance 
of  each  reading. 

Cognitive  time  related  to  test  selection.  Subjects  would  be  given  a  suspicion  state,  i.e.,  a  set  of 
suspected  elements  in  a  system,  and  be  asked  to  select  me  next  test  to  perform. 

All  three  of  the  above  investigations  would  have  to  be  done  over  a  range  of  operational  complexity 
and  process  complexity,  quantified  as  outlined  in  section  IV. 

Cognitive  time  related  to  characteristics  of  alternatives.  Subjects  would  be  given  a  small  set  of 
alternative  actions  and  associated  expectations  and  costs,  and  be  asked  to  choose  a  course  of  action. 

Unfortunately,  studies  one  and  two,  above,  involve  cognitive  functions  which  we  can  only 
presume  are  performed,  and  both  require  the  individual  to  perform  some  response  for  our  benefit 
which  may  involve  further  cognitive  functions  or  may  alter  the  way  the  individual  would  naturally 
respond  to  the  situation.  It  would  be  expected  that  the  results  of  such  studies  would  serve  as  a 
guide  in  then  conducting  more  whole-task  studies,  similar  to  those  used  in  this  research,  to  explore 
cognitive  workload  in  a  realistic  context.  The  three  types  of  cognitive  complexity  developed  here 
may  serve  as  useful  control  variables. 

The  examination  of  complexity  may  also  provide  a  somewhat  broader  or  more  precise 
outlook  upon  the  objectives  which  should  be  sought  to  ease  corrective  maintenance.  To  the 
designer,  a  measure  of  complexity  of  diagnosis  could  have  significance  in  evaluating  alternative 
maintainability  features,  and  someday  the  procurement  of  systems  may  include  specifications 
concerning  ease  of  maintenance  in  terms  which  could  be  so  quantified. 

To  those  concerned  with  maintenance  procedures  and  maintenance  training,  a  consideration 
of  complexity  could  suggest  that  preferred  approaches  to  'ault  isolation  may  not  always  be  those 
which  are  most  powerful,  from  the  standpoint  of  amount  of  information  obtained.  For  example, 
research  might  determine  that  fault  isolation  is  more  reliable  and  timely  when  the  tests  are  selected 
to  yield  less  than  the  maximum  amount  possible,  at  each  stage,  thereby  easing  the  task  of 
interpreting  the  results.  Currently  this  is  a  question  which  is  not  formally  raised,  since  there  has 
been  no  workable  means  for  assessing  the  values  needed.  As  a  result,  maintainers  are  typically 
drilled  to  select  the  most  powerful  tests  available.  On  the  job,  however,  it  may  be  that  the 
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Appendix  A  -  Individual  Technician  Performance  Data 
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Appendix  B  -  Problem  Data 


Appendix  C  -Performance  Data  by  Technician  Group 
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ACTUAL  AND  PROJECTED  PERFORMANCE  DATA 


Appendix  F  -  Interstep  Cognitive  Time  Summary 


COGNITIVE  TIMES  PRIOR  TO  TESTS: 
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COGNITIVE  TIMES  PRIOR  TO  REPLACEMENTS: 

Range  of  Rpimt 
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5  to  15 
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24.4 
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16  to  25 
26  to  35 
36  to  45 
46  to  55 
56  to  107 
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